Video Summarization Techniques Research Articles

A Framework for Video Summarization using Visual Attention Technique

Objectives: To develop an efficient Video Summarization technique that aims to utilize the saliency map for mimicking the human way of selecting the important events in the given video. Methods: This paper proposes Histogram based Weighted Fusion (HWF) algorithm that uses spatial and temporal saliency maps to act as guidance in creating the summary of the video. The spatial saliency score and temporal saliency score obtained from the corresponding saliency maps are fused using the proposed HWF algorithm to obtain the frame level importance score. It tries to depict the visual attention of the human brain when watching a particular video. Findings: The experimental results show that the proposed HWF algorithm performs better than the state-of-the-art methods. Novelty: The use of Histogram intersection and the incorporation of the exponential function as the weight for the combined feature enhance the summarization ability of the proposed model. Keywords: Video Summarization, Saliency Map, Histogram intersection, Contrast sensitivity function, Attention curves

Attention-Based Multi-Layered Encoder-Decoder Model for Summarizing Non-Interactive User-Based Videos

Video summarization extracts the relevant contents from a video and presents the entire content of the video in a compact and summarized form.  User based video summarization, can summarize a video as per the requirement of the user. In this work, a non interactive and a perception-based video summarization technique is proposed that makes use of attention mechanism to capture user’s interest and extract relevant keyshots in temporal sequence from the video content. Here, video summarization has been articulated as a sequence-to-sequence learning problem and a supervised method has been proposed for summarization of the video. Adding layers to the existing network makes it deeper, enables higher level of abstraction and facilitates better feature extraction. Therefore, the proposed model uses a multi-layered, deep summarization encoder-decoder network (MLAVS), with attention mechanism to select final keyshots from the video. The contextual information of the video frames is encoded using a multi-layered Bidirectional Long Short-Term Memory network (BiLSTM) as the encoder. To decode, a multi-layered attention-based Long Short-Term memory (LSTM) using a multiplicative score function is employed. The experiments are performed on the benchmark TVSum dataset and the results obtained are compared with recent works. The results show considerable improvement and clearly demonstrate the efficacy of this methodology against most of the other available state-of-art methods.

Reconstructive network under contrastive graph rewards for video summarization

Video summarization aims to condense video content by extracting pivotal frames or shots. Most existing methods focus on maximizing the intersection between predicted summary and ground truth, overlooking whether users can infer the content of the original video from the summary. Additionally, these approaches heavily rely on annotated data, posing limitations.Therefore, we propose a reconstructive network under contrastive graph rewards for video summarization, comprising a summary generator and a video reconstructor. The summary generator employs graph contrastive learning to distill essential video information to generate summary. Meanwhile, the video reconstructor employs reinforcement learning within an unsupervised training framework to optimize the summary generator, addressing the shortage of annotated video data in summarization tasks. Leveraging reconstruction loss, our approach ensures that predicted summary encapsulate main video content and inter-shots dependencies. Notably, we innovatively devise a mutual information maximization reconstruction reward function to preserve shared information between the summary and the original video, facilitating users in comprehending the original video content. We conduct massive experiments on the TVSum and SumMe datasets, and our network achieved F1 scores of 58.8% and 48.0%, respectively. Experimental results validate the superiority of our method over both state-of-the-art unsupervised and many supervised video summarization techniques.

BNoteHelper: A Note-based Outline Generation Tool for Structured Learning on Video-sharing Platforms

Usually generated by ordinary users and often not particularly designed for learning, the videos on video-sharing platforms are mostly not structured enough to support learning purposes, although they are increasingly leveraged for that. Most existing studies attempt to structure the video using video summarization techniques. However, these methods focus on extracting information from within the video and aiming to consume the video itself. In this article, we design and implement BNoteHelper, a note-based video outline prototype that generates outline titles by extracting user-generated notes on Bilibili, using the BART model fine-tuned on a built dataset. As a browser plugin, BNoteHelper provides users with video overview and navigation as well as note-taking template, via two main features: outline table and navigation marker. The model and prototype are evaluated through automatic and human evaluations. The automatic evaluation reveals that, both before and after fine-tuning, the BART model outperforms T5-Pegasus in BLEU and Perplexity metrics. Also, the results from user feedback reveal that the generation outline sourced from notes is preferred by users over that sourced from video captions due to its more concise, clear, and accurate characteristics but also too general with less details and diversities sometimes. Two features of the video outline are also found to have respective advantages, especially in holistic and fine-grained aspects. Based on these results, we propose insights into designing a video summary from the user-generated creation perspective, customizing it based on video types, and strengthening the advantages of its different visual styles on video-sharing platforms.

A Comparative analysis of Video Summarization techniques for different domains

As technology progresses, a gigantic amount of video data is generated day by day. Pro-cessing such a huge video needs time and requires increased storage and computational power. Sometimes it is convenient for the user to watch a summary or highlight rather than a complete video, which is time-consuming. So, a fully automated solution is required to extract important segments from a video. Researchers have proposed multiple approaches/techniques for summarizing the videos, which resolve the problem of long videos and summarize them according to the video type. This survey and comparative evaluation of video summarizing techniques based on several domains are presented in this paper. Primarily, these methods are classified into different categories based on their methods or techniques used. Then an over-view of some the latest literature is presented with the dataset and evaluation approaches used. The review is also made related to the domain direction and concluded by presenting benefits and difficulties associated with current video summarization techniques

An Evaluation-based Analysis of Video Summarising Methods for Diverse Domains

As technology progresses, a gigantic amount of video data is generated day-by-day. Processing of such a huge video requires time, increased storage, and a computational power. Sometimes it is convenient for the user to watch a summary or highlight rather than watching a complete video, which is a time-consuming task. So, a fully automated solution is required to extract important segments from video. Researchers have proposed multiple approaches / techniques for summarizing the videos which resolve the problem of long videos and summarize them according to the video type. This survey and comparative evaluation of video summarizing techniques based on several domains are presented in this study. Primarily, these methods are classified into different categories based on their methods or techniques used. Furthermore, an overview of some of the latest literature is presented with the dataset and the evaluation approaches used. The review is also made related to the domain direction, and is concluded by presenting the benefits and difficulties associated with the current video summarization techniques.

A comprehensive study of automatic video summarization techniques

A comprehensive study of automatic video summarization techniques

FIAS3: Frame Importance-Assisted Sparse Subset Selection to Summarize Wireless Capsule Endoscopy Videos

Wireless capsule endoscopy (WCE) is a recently developed tool that allows for the painless and non-invasive examination of the entire gastrointestinal (GI) tract. The microcamera captures a large number of redundant frames for each WCE examination such that a video summarization technique is needed to assist in diagnosis. However, prevalent methods of summarizing WCE videos focus only on the representativeness of the frames owing to a lack of high-level information on their importance. This paper develops a Frame Importance-Assisted Sparse Subset Selection model, called FIAS3, to integrate the high-level frame importance from networks into a sparse subset selection model. The FIAS3 is optimized under three constraints: 1) a frame importance matrix to help pay more attention to important frames, 2) a sparsity constraint to make video summaries more compact, and 3) a similarity-inhibiting constraint to reduce redundancy. The results of experiments on a public dataset demonstrated that our FIAS3 outperforms other methods of summarizing WCE videos. Specifically, its coverage and video reconstruction error were 92% and 0.143, respectively, at a 90% compression ratio, recording respective at least 16.9% and 0.031 improvements over other methods. The results of generalization experiments showed that FIAS3 also achieves competitive results on private datasets.

A Player-Specific Framework for Cricket Highlights Generation Using Deep Convolutional Neural Networks

Automatic ways to generate video summarization is a key technique to manage huge video content nowadays. The aim of video summaries is to provide important information in less time to viewers. There exist some techniques for video summarization in the cricket domain, however, to the best of our knowledge our proposed model is the first one to deal with specific player summaries in cricket videos successfully. In this study, we provide a novel framework and a valuable technique for cricket video summarization and classification. For video summary specific to the player, the proposed technique exploits the fact i.e., presence of Score Caption (SC) in frames. In the first stage, optical character recognition (OCR) is applied to extract text summary from SC to find all frames of the specific player such as the Start Frame (SF) to the Last Frame (LF). In the second stage, various frames of cricket videos are used in the supervised AlexNet classifier for training along with class labels such as positive and negative for binary classification. A pre-trained network is trained for binary classification of those frames which are attained from the first phase exhibiting the performance of a specific player along with some additional scenes. In the third phase, the person identification technique is employed to recognize frames containing the specific player. Then, frames are cropped and SIFT features are extracted from identified person to further cluster these frames using the fuzzy c-means clustering method. The reason behind the third phase is to further optimize the video summaries as the frames attained in the second stage included the partner player’s frame as well. The proposed framework successfully utilizes the cricket videoo dataset. Additionally, the technique is very efficient and useful in broadcasting cricket video highlights of a specific player. The experimental results signify that our proposed method surpasses the previously stated results, improving the overall accuracy of up to 95%.

A review on video summarization techniques

The exponential growth of technology has resulted in a profusion of advanced imaging devices and eases internet accessibility, leading to an increase in the creation and use of multimedia content. Analyzing representative or meaningful information from such massive data is a time-consuming task that impacts the efficiency of various video processing applications, including video searching, retrieval, indexing, sharing, and many more. In literature, numerous video summarization techniques which extract key-frames or key-shots from the original video to generate a concise yet informative summary have been proposed to address these issues. This paper presents a discussion of the state-of-the-art video summarization techniques along with limitations and challenges. The paper examines summarization techniques in a holistic manner based upon the distinct attributes of evolving video data types on the basis of parameters such as the number of views, dimensions, modality, and content. Such a categorization framework enables us to critically analyze the recent progress, future directions, limitations, datasets, application domains etc., in a better comprehensible manner.

Creating Video Visual Storyboard with Static Video Summarization using Fractional Energy of Orthogonal Transforms

The overwhelming number of video uploads and downloads has made it incredibly difficult to find, gather, and archive videos. A static video summarization technique highlights an original video's significant points through a set of static keyframes as a video visual storyboard. The video visual storyboards are created as static video summaries that solve video processing-related issues like storage and retrieval. In this paper, a strategy for effectively summarizing static videos using the feature vectors, which are fractional coefficients of the transformed video frames, is proposed and evaluated. Four popular orthogonal transforms are deployed for generating feature vectors of video frames. The fractional coefficients of transformed video frames taken as 25 percent, 6.25 percent, and 1.5625 percent of full 100 percent transformed coefficients are considered to form video visual storyboards. The proposed method uses the benchmark video datasets Open Video Project (OVP) and SumMe to validate the performance, containing user summaries (storyboards). These video summaries created using the proposed method are evaluated using percentage accuracy and matching rate.

Person Retrieval in Video Surveillance Using Deep Learning–Based Instance Segmentation

Video surveillance systems are deployed at many places such as airports, train stations, and malls for security and monitoring purposes. However, it is laborious to search for and retrieve persons in multicamera surveillance systems, especially with cluttered backgrounds and appearance variations among multiple cameras. To solve these problems, this paper proposes a person retrieval method that extracts the attributes of a masked image using an instance segmentation module for each object of interest. It uses attributes such as color and type of clothes to describe a person. The proposed person retrieval system involves four steps: (1) using the YOLACT++ model to perform pixelwise person segmentation, (2) conducting appearance-based attribute feature extraction using a multiple convolutional neural network classifier, (3) employing a search engine with a fundamental attribute matching approach, and (4) implementing a video summarization technique to produce a temporal abstraction of retrieved objects. Experimental results show that the proposed retrieval system can achieve effective retrieval performance and provide a quick overview of retrieved content for multicamera surveillance systems.

A survey of recent work on video summarization: approaches and techniques

The volume of video data generated has seen an exponential growth over the years and video summarization has emerged as a process that can facilitate efficient storage, quick browsing, indexing, fast retrieval and quick sharing of the content. In view of the vast literature available on different aspects of video summarization approaches and techniques, a need has arisen to summarize and organize various recent research findings, future research focus and trends, challenges, performance measures and evaluation and datasets for testing and validations. This paper investigates into the existing video summarization frameworks and presents a comprehensive view of the existing approaches and techniques. It highlights the recent advances in the techniques and discusses the paradigm shift that has occurred over the last two decades in the area, leading to considerable improvement. Attempts are made to consolidate the most significant findings right from the basic summarization structure to the classification of summarization techniques and noteworthy contributions in the area. Additionally, the existing datasets categorized domain-wise for the purpose of video summarization and evaluation are enumerated. The present study would be helpful in: assimilating important research findings and data for ready reference, identifying groundwork and exploring potential directions for further research.

A Survey on various Video Summarization Techniques

In this fast-moving world people don’t have the time to watch lengthy videos, so it would be convenient for them if they could access short summaries of these videos and be able to acquire more information by watching summarized videos. A study of the different methods used for video summarization over the years i.e., extracting important segments from a video to produce short concise summaries that are representative of the original video is presented.

An efficient video summarization for surveillance system using normalized k-means and quick sort method

There has been a vast augmentation in quantity of Video Content (VC) generated amid the last some years. The Video Summarization (VS) approach is introduced for managing the VC. Prevailing VS techniques have endeavored to render the VS but the systems have Execution Time (ET) as well as condensing the video's content in domain specific manner. To triumph over such disadvantages, this paper proposed efficient VS for surveillance system using normalized k-means along with quick sort method. The proposed technique comprises eight stages, like split video into frames, pre-sampling, provide ID number, feature extraction, Feature Selection (FS), clustering, extract frames, video summary. Initially, the video frames are pre-sampled utilizing the proposed Three Step Cross Searching Algorithm (TSCS) technique. Then, give the ID number for every frame. Next, the features are extracted as of the frames. Then, the necessary features are selected using Entropy based Spider Monkey Algorithms (ESMA). In next stage, the features are grouped using Normalized K-Means (N-Kmeans) algorithm for identifying best candidate frames. Then select the minimum distance value based cluster set is the Key Frame (KF) selection. At last, the video is orderly summarized using quick sort method. Finally, in experimental evaluation the proposed work is compared with the prevailing methods. The proposed VS gave better outcome than the existing approaches.

Summarization of Wireless Capsule Endoscopy Video Using Deep Feature Matching and Motion Analysis

Conventional Wireless capsule endoscopy (WCE) video summary generation techniques apprehend an image by extracting hand crafted features, which are not essentially sufficient to encapsulate the semantic similarity of endoscopic images. Use of supervised methods for extraction of deep features from an image need an enormous amount of accurate labelled data for training process. To solve this, we use an unsupervised learning method to extract features using convolutional auto encoder. Furthermore, WCE images are classified into similar and dissimilar pairs using fixed threshold derived through large number of experiments. Finally, keyframe extraction method based on motion analysis is used to derive a structured summary of WCE video. Proposed method achieves an average F-measure of 91.1% with compression ratio of 83.12%. The results indicate that the proposed method is more efficient compared to existing WCE video summarization techniques.

A Study on Multi-view Video Summarization Techniques

With the recent developments in multimedia and their applications in day-to-day life, the summarization of video content is a growing and popular field. The main objective of video summarization is to abstract the important events, or objects from a video to provide an easily interpretable summary. The video-surveillance is a crucial security mechanism of many public and private places like hospitals, universities, banks, etc. These videos provide important information about any intruder or invaders. Although this information is of critical importance, it takes a lot of time and effort to go through hours of long videos in order to find out the significant events. Here, multi-view video summarization comes to rescue as it extracts the most useful events out of the videos and presents it to the user. The multi-view video summarization system provides an advantage over a single view video summarization system by exploiting the complicated intra-view and inter-view correlations, thereby presenting an effective summary of multiple camera views and enriching the video content. It explores various dimensions of the same event and collectively may give more complete information. In this paper, we present a study on multi-view video summarization techniques. We examine work done on various methods of video summarization for videos consisting of many views of the same location as captured by the surveillance cameras positioned at different angles. The final summarized video can be used for the identification of suspicious events, an inspection of post-accidental events, thefts in public places, etc., which will be supportive to the crime department for investigation purposes.

Social media video summarization using multi-Visual features and Kohnen's Self Organizing Map

Social networking tools such as Facebook, YouTube, Twitter, and Instagram, are becoming major platforms for communication. YouTube as one of the primary video sharing platform serves over 100 million distinct videos, 300 hours of videos are uploaded on YouTube every minute along with textual data. This massive amount of multimedia data needs to be managed with high efficiency, the irrelevant and redundant data needs to be removed. Video summarization ideals with the problem of redundant data in a video. A summarized video contains the most distinct frames which are termed as key frames. Most of the research work on key frames extraction considers only a single visual feature which is not sufficient for capturing the full pictorial details and hence affecting the quality of video summary generated. So there is a need to explore multiple visual features for key frames extraction. In this research work a key frame extraction technique based upon fusion of four visual features namely: correlation of RGB color channels, color histogram, mutual information and moments of inertia is proposed. Kohonen Self Organizing map as a clustering approach is used to find the most representative frames from the list of frames coming after fusion. Useless frames are discarded and frames having maximum Euclidean distance within a cluster are selected as final key frames. The results of the proposed technique are compared with the existing video summarization techniques: User generated summary, Video SUMMarization (VSUMM), and Video Key Frame Extraction through Dynamic Delaunay Clustering (VKEDDCSC) which shows a considerable improvement in terms of fidelity and Shot Reconstruction Degree (SRD) score.

DeepReS: A Deep Learning-Based Video Summarization Strategy for Resource-Constrained Industrial Surveillance Scenarios

The exponential growth in the production of video contents in different industries causes an urgent need for effective video summarization (VS) techniques, in order to get an optimal storage and preservation of key information in the video. Compared to other domains, industrial videos are more challenging to process, as they usually contain diverse and complex events, which make their online processing a difficult task. In this article, we introduce an online system for intelligent video capturing, coarse and fine redundancy removal, and summary generation. First, we capture video data through resource-constrained devices in an industrial Internet of Things network, equipped with vision sensors and apply coarse redundancy removal through the comparison of low-level features. Second, we transmit the resulting frames to the cloud for detailed analysis, where sequential features are extracted for the selection of candidate keyframes. Finally, we refine the candidate keyframes in order to discriminate those with maximum information as part of the summary. The key contributions of this article include the coarse and fine refining of video data implemented over resource-restricted devices and the presentation of important data in the form of a summary. Experiments 1 1 [Online]. Available: https://github.com/tanveer-hussain/DeepRes-Video-Summarization . over publicly available datasets evince a 0.3-unit increase in the F1 score when compared to state-of-the-art and with reduced time complexity. Furthermore, we provide convincing results on our newly created dataset in an industrial environment, which is made publicly available for the research community along with its labeled ground truth.

Video Summarization Based on Gaussian Mixture Model and Kernel Support Vector Machine for Forest Fire Detection

Exponential growth in the generation of multimedia data especially videos resulted to the development of video summarization concept. The summary of the videos offers a collection of frames which precisely define the video content in a considerably compacted form. Video summarization models find its applicability in various domains especially surveillance. This paper intends to develop a video summarization technique for the application of forest fire detection. The proposed method involves a set of processes namely convert frames, key frame extraction, feature extraction and classification. Here, a Merged Gaussian Mixture Model (MGMM) is applied for the process of extracting key frames and kernel support vector machine (KSVM) is employed for classifying a frame into normal frame and forest fire frame. The simulation analysis is performed on the forest fire video files from FIRESENSE database and the results are assessed under several dimensions. The final outcome proves the efficiency of the presented MGMM-KSVM model in a considerable way.